Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/higher order mesh particle interface #265

Open
wants to merge 71 commits into
base: main
Choose a base branch
from

Conversation

will-saunders-ukaea
Copy link
Contributor

@will-saunders-ukaea will-saunders-ukaea commented Dec 2, 2024

Description

  • Adds Initial support for particles on curved meshes.
  • Adds faster function evaluation method via Bary interpolation evaluation (required for the curved mesh support).

Type of change

  • New feature (non-breaking change which adds functionality)

Testing

  • Added curved mesh for testing XMaps and inverse XMaps on curved meshes.
  • Added Bary evaluation tests for function evaluation/function derivative evaluation.
  • Added Integration test for particle advection on curved meshes.

@will-saunders-ukaea will-saunders-ukaea added Awaiting Review PR waiting to be reviewed. and removed work in progess Draft PR - not to merge! labels Jan 7, 2025
@jwscook
Copy link
Member

jwscook commented Jan 8, 2025

@oparry-ukaea to review bary and geometry transport
@JamesEdgeley review the particle_cell_mapping files.
@seimtpow to review the nektar_interface code

please all confirm that the tests pass.

@JamesEdgeley
Copy link
Contributor

All tests pass using gcc 13.3 and adaptivecpp 24.02.0

With oneapi 2025.0.0
The following tests FAILED:
8 - /CompositeInteractionAllD.Intersection/ (ILLEGAL)
9 - /CompositeInteractionAllD.Reflection/ (ILLEGAL)
16 - ParticleFunctionEvaluationSubGroup.2D (ILLEGAL)
17 - ParticleFunctionEvaluationSubGroup.3DContField (ILLEGAL)
18 - ParticleFunctionEvaluationSubGroup.3DDisContFieldHex (ILLEGAL)
19 - ParticleFunctionEvaluationSubGroup.3DDisContFieldPrismTet (ILLEGAL)
20 - ParticleFunctionProjectionSubGroup.2D (ILLEGAL)
21 - ParticleFunctionProjectionSubGroup.3DContField (ILLEGAL)
22 - ParticleFunctionProjectionSubGroup.3DDisContFieldHex (ILLEGAL)
23 - ParticleFunctionProjectionSubGroup.3DDisContFieldPrismTet (ILLEGAL)
24 - ParticleFunctionEvaluation3D.ContField (ILLEGAL)
25 - ParticleFunctionEvaluation3D.DisContFieldHex (ILLEGAL)
28 - BaryInterpolation.Evaluation3DDisContFieldHex (Failed)
29 - BaryInterpolation.Evaluation3DDisContFieldPrismTet (Failed)
37 - ParticleFunctionProjection3DBasisEval.ContField (ILLEGAL)
38 - ParticleFunctionProjection3D.DisContFieldHex (ILLEGAL)
75 - /ParticleGeometryInterface.LocalMapping3D/ (ILLEGAL)
83 - SimpleSOLTest.2DWithParticles (ILLEGAL)
84 - CompositeInteraction.MASTUReflection (ILLEGAL)
85 - ParticleGeometryInterface.Advection2D (ILLEGAL)
86 - /ParticleAdvection3D.Advection3D/ (ILLEGAL)
87 - ParticleAdvection3D.Torus (ILLEGAL)
91 - ParticleFunctionProjectionOrder3D.DisContFieldHex (ILLEGAL)
92 - ParticleFunctionProjectionOrder3D.DisContFieldPrismTet (ILLEGAL)
93 - ParticleFunctionProjectionOrder3D.ContFieldHex (ILLEGAL)
94 - ParticleFunctionProjectionOrder3D.ContFieldPrismTet (ILLEGAL)
95 - Electrostatic2D3V.TwoStream (ILLEGAL)
99 - HWTest.Coupled2Din3DHWMassCons (ILLEGAL)

Are they passing for you @oparry-ukaea @seimtpow ?

@oparry-ukaea
Copy link
Contributor

With oneapi 2025.0.0 The following tests FAILED:
8 - /CompositeInteractionAllD.Intersection/ (ILLEGAL)
9 - /CompositeInteractionAllD.Reflection/ (ILLEGAL)
...

Are you getting 'KILLED BY SIGNAL: 4 (Illegal instruction)'?

@JamesEdgeley
Copy link
Contributor

With oneapi 2025.0.0 The following tests FAILED:
8 - /CompositeInteractionAllD.Intersection/ (ILLEGAL)
9 - /CompositeInteractionAllD.Reflection/ (ILLEGAL)
...

Are you getting 'KILLED BY SIGNAL: 4 (Illegal instruction)'?

Thread 9 "unitTests" received signal SIGILL, Illegal instruction. (from gdb)

@oparry-ukaea
Copy link
Contributor

Sounds like the opencl bug Will found before Christmas (see the standups slack)
You might need to
export CL_CONFIG_CPU_TARGET_ARCH=core-avx2
or if that doesn't work
export CL_CONFIG_CPU_TARGET_ARCH=corei7-avx

@JamesEdgeley
Copy link
Contributor

Sounds like the opencl bug Will found before Christmas (see the standups slack) You might need to export CL_CONFIG_CPU_TARGET_ARCH=core-avx2 or if that doesn't work export CL_CONFIG_CPU_TARGET_ARCH=corei7-avx

Ah I missed that - second one works for me. Thanks!

Copy link
Contributor

@oparry-ukaea oparry-ukaea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed

  • CMakeLists.txt
  • Everything in include/nektar_interface/bary_interpolation
  • Everything in include/nektar_interface/geometry_transport
  • All files test/unit/nektar_interface/test_particle_geometry_interface*.cpp

@@ -0,0 +1,4082 @@
<?xml version="1.0" encoding="utf-8" ?>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just commit the compressed version of the mesh xml?

* @param[in] coord The evauation point in the dimension of interest.
* @param[in] z_values A length num_phys array containing the quadrature
* points.
* @param[in] z_values A length num_phys array containing the quadrature
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @param[in] z_values A length num_phys array containing the quadrature
* @param[in] bw_values A length num_phys array containing the quadrature

* @param[in] coord The evauation points, size N, in the dimension of interest.
* @param[in] z_values A length num_phys array containing the quadrature
* points.
* @param[in] z_values A length num_phys array containing the quadrature
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @param[in] z_values A length num_phys array containing the quadrature
* @param[in] bw_values A length num_phys array containing the quadrature

static_assert(!Newton::local_memory_required<
Newton::MappingQuadLinear2DEmbed3D>::required,
"Did not expect local memory to be required for this Newton "
"implemenation");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"implemenation");
"implementation");

@@ -79,22 +104,13 @@ class PackedGeom2D {

// curve of the edge
auto curve = seg_geom->GetCurve();
ASSERTL0(curve == nullptr, "Not implemented for curved edges");
// A curve with n_points = -1 will be a taken as non-existant.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// A curve with n_points = -1 will be a taken as non-existant.
// A curve with n_points = -1 will be a taken as non-existent.

Comment on lines +34 to +38
auto xmap = geom->GetXmap();
const bool linear = (xmap->GetBasisNumModes(0) == 2) &&
(xmap->GetBasisNumModes(1) == 2) &&
(xmap->GetBasisNumModes(2) == 2);
if (linear) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
auto xmap = geom->GetXmap();
const bool linear = (xmap->GetBasisNumModes(0) == 2) &&
(xmap->GetBasisNumModes(1) == 2) &&
(xmap->GetBasisNumModes(2) == 2);
if (linear) {
if (geometry_is_linear(geom) {

Use geometry_is_linear from utility_geometry.hpp?

@@ -341,7 +415,8 @@ TEST(ParticleGeometryInterface, CoordinateMapping3D) {
xi1[dimx] = dh_xi.h_buffer.ptr[dimx];
eta0[dimx] = dh_eta.h_buffer.ptr[dimx];
}
geom->GetXmap()->LocCollapsedToLocCoord(eta0, xi0);
// geom->GetXmap()->LocCollapsedToLocCoord(eta0, xi0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// geom->GetXmap()->LocCollapsedToLocCoord(eta0, xi0);

@@ -367,7 +442,9 @@ TEST(ParticleGeometryInterface, CoordinateMapping3D) {
eta1[dimx] = dh_eta.h_buffer.ptr[dimx];
xi0[dimx] = dh_xi.h_buffer.ptr[dimx];
}
geom->GetXmap()->LocCoordToLocCollapsed(xi0, eta0);
// geom->GetXmap()->LocCoordToLocCollapsed(xi0, eta0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// geom->GetXmap()->LocCoordToLocCollapsed(xi0, eta0);

lambda_test_wrapper(geom, geom_test);
} else {
// Unknown shape type
ASSERT_TRUE(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ASSERT_TRUE(false);
FAIL() << "Unknown shape type.";

?

GeometryInterface::Triangle geom_test{};
lambda_test_wrapper(geom, geom_test);
} else {
ASSERT_TRUE(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ASSERT_TRUE(false);
FAIL() << "Expected only triangles or quads but found a different shape type.";

?

@oparry-ukaea
Copy link
Contributor

All tests passing for me with GCC 11.3.0 / AdaptiveCpp 24.06.0

With OneAPI 2024.1.0/ DPC++ 2022.1.0 I get intermittent failures of
BaryInterpolation.Evaluation3DDisContFieldHex and BaryInterpolation.Evaluation3DDisContFieldPrismTet

It's always very marginal, e.g.
The difference between to_test and to_test_stride is 2.6756374893466273e-14, which exceeds 1.0e-14...
The difference between to_test and to_test_stride is 1.1657341758564144e-14, which exceeds 1.0e-14...
etc.

Could/should the tolerance be relaxed a bit?

Copy link
Contributor

@JamesEdgeley JamesEdgeley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All tests pass for me bar the one mentioned.
Reviewed everything in a particle_cell_mapping directory

x0, x1, x2, num_phys0, num_phys1, num_phys2, physvalsv.data(),
div_space.data(), z0v.data(), z1v.data(), z2v.data(), bw0v.data(),
bw1v.data(), bw2v.data(), stride);
EXPECT_NEAR(to_test, to_test_stride, 1.0e-14);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tolerance is a little too low for the some of the tests to pass for me with oneapi (1.0e-13 would be fine). Could it be set using the tol argument rather than hard-coded here?

get_all_elements_3d(particle_mesh_interface->graph, geoms);
for (auto gx : geoms) {
std::pair<int, std::shared_ptr<Nektar::SpatialDomains::Geometry3D>>
tmp = {gx.first, gx.second};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is tmp created instead of just using gx?

this->num_bytes_local_memory =
std::max(this->num_bytes_local_memory, s);
};
for (auto &geom : geoms_local) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A structured binding might be good here

}

inline void free_data_v(void *data_host) {
Generic3D::DataHost *h_data = static_cast<Generic3D::DataHost *>(data_host);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the DataHost that was created earlier at *data_host with the placement new get destroyed properly? This function frees the data in the buffer but I'm wondering if the memory occupied by the DataHost itself gets freed.

@seimtpow
Copy link
Contributor

seimtpow commented Jan 9, 2025

unitTests pass for me with clang 18.1.18 and AdaptiveCPP (commit #fca48591), works with OpenMP and CUDA backends - had to remove some EQTYPE elements from various conditions*.xml test resource files (This is caused by issue #258, and should be fixed when #259 is merged).

std::unique_ptr<BufferDeviceHost<char>> dh_data;
std::unique_ptr<BufferDeviceHost<std::byte>> dh_data;
/// The data required to perform newton iterations for each geom on the host.
std::unique_ptr<BufferHost<std::byte>> h_data;
Copy link
Contributor

@seimtpow seimtpow Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe missing something, but I don't like the buffer of bytes/void pointer - doesn't seem necessary as everything is known at compile time.

For example each NEWTON_TYPE class could add e.g.

using data_type = DataDevice;

and then we could get that via typename NEWTON_TYPE::data_type and just make dh_data a pointer to the correct type - it could work something like this

 template <class T> struct A {
  int do_something(void *data) {
    auto &under = static_cast<T &>(*this);
    return under.do_something_v(static_cast<typename T::data_type *>(data));
  }
  void write(void *data) {
    auto &under = static_cast<T &>(*this);
    under.write_v(static_cast<typename T::data_type *>(data));
  }
};

struct B : A<B> {
  using data_type = double;
  int do_something_v(data_type *data) { return (int)*data; }
  void write_v(data_type *data) { *data = 1.0; }
};

struct C : A<C> {
  using data_type = int;
  int do_something_v(data_type *data) { return *data + 1; }
  void write_v(data_type *data) { *data = 1; }
};

template <typename K> struct Xthing {
  using KType = typename K::data_type;
  KType *ptr;
  Xthing() { ptr = new KType; }
  ~Xthing() { delete ptr; }
};

Also not sure the CRTP is needed AFAICS you should be able to just use static functions if there is no state in the derived classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Awaiting Review PR waiting to be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants